Setting up Billing Exports: BigQuery

In this lesson, you will learn about exporting billing data to BigQuery (BQ).

We'll cover the following

Previously, GCP used to also support file export options, but it is now deprecated by GCP.

In this lesson, we will create a sink to export billing data into the BigQuery dataset. This will allow us to query our billing data. For a smaller dataset, this doesn’t make much sense. However, when an organization is large enough to generate more than GBs of billing data, BQ analytics make much more sense to learn the costs and spending patterns.

Exporting billing data#

To enable the export, follow the steps given below.

  1. Main menu > Billing

  2. Select the primary billing account. Click on Billing export.

  3. Select the “Bigquery export” option.

Click on "Billing export"
Click on "Billing export"
  1. By default it is disabled you need to enable the billing.
  2. Click on the “daily cost detail” section “Edit Settings” buttons.

The standard practice is to keep the historical data or bookkeeping data in a separate project not used by development resources. This demo is up to you By now, you should have a fair understanding of creating a project and attaching it to a billing account.

  1. GCP will look for a dataset in the BQ. We don’t have any so create one using the “Go To Bigquery” button.

  2. This will open up the BQ dashboard.

widget
widget
  1. In the left pane, click on the project name at the bottom. It will show the “Create Dataset” button on the right side.
Click on the project and then click the "CREATE DATASET" button.
Click on the project and then click the "CREATE DATASET" button.

  1. Once you click the “CREATE DATASET” button, it will open up one form. Fill in the form to create the dataset.

  2. We will select “Never” for data expiration as we wanted to store this data forever.

Name the dataset and click the Create dataset button.
Name the dataset and click the Create dataset button.

Once we have the dataset ready, go back to the export dashboard and refresh. Click on “Edit settings” and select the project in which the dataset is present.

The dataset will auto-populate this time. Click on the “Save” button.

This will create a table of “Daily cost detail” in the BQ dataset.

The data set will auto populate based on selected project. Click on "SAVE" button.
The data set will auto populate based on selected project. Click on "SAVE" button.

The SKU pricing of GCP billing keeps updating. If you want to store the SKU pricing in the same dataset:

  1. Click on the “Edit Settings” button under the “Pricing” section.
  2. Select the dataset’s project.
  3. Enable “Bigquery Data transfer API” if not enabled already.
  4. Then click Save.
Confirm whether the Data Transfer API is enabled.
Confirm whether the Data Transfer API is enabled.

The export will create 2 tables in the dataset. You can get the schema here.

Both daily cost details and pricing details export should be enabled now.
Both daily cost details and pricing details export should be enabled now.

Cost#

For billing exports:

  • Loading data into the designated dataset is free.

  • When exporting and analyzing cloud billing data with BigQuery, the associated cost will depend on the amount of data you stream, store, and query.

You will see the BQ cost structure in the Big Data section. Once we have the billing data, it will be exported to BQ. Since the generated data will be minimal, we will be charged a negligible amount.

Until now, we have been using GCP through UI only, but this is not always how we work in production. In the next chapter, we will look at the GCP command-line interface. We will install Cloud SDK and look at some of the frequently used commands.

Budgets and Alerts

Quiz